(1) Field of the Invention
The present invention relates to a surroundings monitoring apparatus which generates a synthetic image of the vehicle surroundings using an image captured by a camera.
(2) Description of the Related Art
Conventionally, there are surroundings monitoring apparatuses using image synthesis technology, which generate a synthetic image by deforming an image captured by a camera as if it were taken from a virtual viewpoint, and a synthetic image by synthesizing images captured by a plurality of cameras.
As an example of such a surroundings monitoring apparatus, a surroundings monitoring apparatus is disclosed which captures images of the surroundings of a vehicle by a plurality of cameras and deforms the captured images so as to generate the image seen from above the surroundings of the vehicle (for example, refer to Patent Documents 1 to 3).
FIG. 1A is a block diagram of the conventional surroundings monitoring apparatus, and FIG. 1B shows an example of a synthetic image, which are described in Patent Document 2. In the conventional surroundings monitoring apparatus shown in FIG. 1A, image memories 7-1 to 7-8 temporarily store images captured by the cameras 3-1 to 3-8 at a timing controlled by the timing generator 9 and transmitted to the image memories 7-1 to 7-8 after being converted by the A/D converters 5-1 to 5-8. A combination of an address counter 13-1, a vertical address look-up table (LUT) 13-2, a horizontal address LUT 13-3, and a memory selector LUT 13-4 in the video deforming and synthesizing unit 13 generate the image coordinates for the captured image which correspond to each pixel of the synthetic image, in other words, an address in each of the image memories 7-1 to 7-8. Then, a memory selector 13-5 reads the image in the image memories. Using this surroundings monitoring apparatus, it is possible to generate a synthetic image from the images captured by a plurality of cameras 3-1 to 3-8, as if it were taken from above the vehicle as shown in FIG. 1B, and display the generated synthetic image on a monitor 17, 17-1 after being converted by the D/A converter 13-6. In the case where the surroundings of the vehicle is monitored using such a surroundings monitoring apparatus, it is possible to check, on a screen, a blind spot area which can not be directly seen from a driving seat. Thus, there are advantages, such as avoiding a collision with surrounding objects and facilitating driving operation. In particular, the surroundings monitoring apparatus including the aforementioned look-up tables can generate images by calculating a correspondence between a captured image and a synthetic image in the coordinates beforehand, storing the results in the look-up tables, and referring to the look-up tables in execution. Therefore, there is an advantage that the calculation load in execution is small.
However, in the conventional configuration, in the case where an error occurs between the position of a camera installed in a vehicle and a designed value after creating look-up tables based on the camera installation position in design, or in the case where the position of the camera installed in the vehicle or a location of the vehicle is displaced after creating look-up tables based on the position of the camera measured after installing the camera in the vehicle, the captured image in which the position of the camera (including the direction) is displaced from the position of the camera at the time of creating the look-up tables is inputted. As a result, there is a problem that the resulting synthetic image is distorted. For example, the positions of the surrounding objects in the synthetic image, such as a road surface and other vehicles are displaced with respect to the synthetic image at the time of creating the look-up tables, and the joining areas of the images captured by the cameras which are in the synthetic image become discontinuous. In particular, in the case where graphics which show a reference position relative to the vehicle or a reference distance to the vehicle are superimposed on the synthetic image in which the positions of the surrounding objects are displaced from the actual positions, the superimposing is not preferable because the position relationship between the surrounding objects and the vehicle may be misunderstood. For the description hereinafter, a camera position at the time of creating a look-up table is referred to as a “reference camera position”, an image synthesis parameter such as a look-up table generated with respect to the reference camera position is referred to as a “reference image synthesis parameter”, and an image which is generated using an image captured in the reference camera position and the reference image synthesis parameter is referred to as a “reference synthetic image”.
As a method for correcting the distortions in the synthetic image caused by such position displacement of the camera, a method called the camera calibration method is known which measures an inner parameter, a camera position after installing the camera or the like (for example, refer to Non-patent Document 1).
Even in the case where the camera installation position is displaced, the use of this camera calibration method makes it possible to generate images with almost no position displacement with respect to the reference synthetic image, by measuring a position where the camera is installed and recalculating the look-up tables.    Patent Document 1: Japanese Laid-Open Patent Application No. 58-110334    Patent Document 2: Japanese Laid-Open Patent Application No. 11-78692    Patent Document 3: Japanese Patent No. 3286306 Publication    Non-patent Document 1: “An Efficient and Accurate Camera Calibration Technique for 3D Machine Vision”, Roger Y. Tsai, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Miami Beach, Fla., 1986, pages 364-374.
However, for camera calibration, there exists another problem in that a photographic subject with a known size is required and the calculation load for calculating a position of a camera and for recalculating a look-up table is heavy.
For example, when cameras are installed on vehicles in an assembly plant of the vehicles, even in the case where vehicles to be assembled are the same model, the height or inclination of the vehicles varies depending on differences in the weight of the vehicles with and without optional equipment and differences in the tire size of the vehicles. Therefore, in the case where a reference camera position and a look-up table which are common to the same model are used, even when the cameras are installed in the vehicle as specified by the design, the positions of the cameras are displaced with respect to the surrounding objects. Because of this, as it is necessary to correct the displacement of the cameras one-by-one with the camera calibration method and the like, it takes time to make such correction.
In addition, although it is conceivable to use another method for recalculating the look-up tables after measuring a location displacement of the vehicle by a sensor and the like, there is also another problem that the calculation load is heavy. In particular, as the conventional surroundings monitoring apparatus configured using the look-up tables has an advantage in its small calculation load in execution, the heavy calculation load caused by the correction may lose its advantage.
In addition, other possible methods for correcting distortion in a synthetic image caused by a displacement in the position of a camera includes preparing plural look-up tables appropriate for the degree of the camera position displacement, and selecting one of the look-up tables in execution. However, there is another problem that the amount of memory for storing such look-up tables becomes enormous. Furthermore, there is another problem that an error in a synthetic image may become larger depending on the correction methods.
Thus, the present invention has been conceived in view of the aforementioned circumstances, and aims at providing a surroundings monitoring apparatus and a surroundings monitoring method which can reduce distortions, on a displayed image, which occur due to a position displacement of a camera in the case where the position of the camera is displaced with respect to the reference camera position used for calculating a parameter for synthesizing images, such as a look-up table.